Recommending edges via importance aware machine learned model

ABSTRACT

In an example embodiment, a single machine learned model that allows for ranking of entities across all of the different combinations of node types and edge types is provided. The solution calibrates the scores from Edge-FPR models to a single scale. Additionally, the solution may utilize a per-edge type multiplicative factor dictated by the true importance of an edge type, which is learned through a counterfactual experimentation process. The solution may additionally optimize on a single, common downstream metric, specifically downstream interactions that can be compared against each other across all combinations of node types and edge types.

TECHNICAL FIELD

The present disclosure generally relates to technical problemsencountered in machine learning. More specifically, the presentdisclosure relates to recommending edges via an importance-aware machinelearned model.

BACKGROUND

The rise of the Internet has occasioned two disparate yet relatedphenomena: the increase in the presence of online networks, with theircorresponding user profiles visible to large numbers of people, and theincrease in the use of these online networks to provide content. Onlinenetworks generally provide users with the ability to create a userprofile and then to connect their user profile to other user profiles (aprocess commonly referred to as “linking” or “friending”). Theconnections between users may be used to form a social network graph,with users represented by nodes in the graph and the connections betweenusers represented by edges between the nodes.

The social network graph may be used to recommend other users that auser may be interested in connecting with. There may be instances,however, where it would be helpful to be able to recommend other typesof entities in an online network, other than just users. For example, asocial networking service may wish to recommend companies or hashtagsthat a user may be interested in following, or newsletters, groups, orevents that a user may be interested in subscribing to.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the technology are illustrated, by way of exampleand not limitation, in the figures of the accompanying drawings.

FIG. 1 is a block diagram illustrating a client-server system, inaccordance with an example embodiment.

FIG. 2 is a block diagram showing the functional components of an onlinenetwork, including a data processing module referred to herein as asearch engine, for use in generating and providing search results for asearch query, consistent with some embodiments of the presentdisclosure.

FIG. 3 is a block diagram illustrating an application server module ofFIG. 2 in more detail, in accordance with an example embodiment.

FIG. 4 is a flow diagram illustrating a method for utilizing a pluralityof machine-learned models, in accordance with an example embodiment.

FIG. 5 is a block diagram illustrating a screen capture of a graphicaluser interface, in accordance with an example embodiment.

FIG. 6 is a block diagram illustrating a software architecture, inaccordance with an example embodiment.

FIG. 7 illustrates a diagrammatic representation of a machine in theform of a computer system within which a set of instructions may beexecuted for causing the machine to perform any one or more of themethodologies discussed herein, according to an example embodiment.

DETAILED DESCRIPTION

Overview

The present disclosure describes, among other things, methods, systems,and computer program products that individually provide variousfunctionality. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the various aspects of different embodimentsof the present disclosure. It will be evident, however, to one skilledin the art, that the present disclosure may be practiced without all ofthe specific details.

In an example embodiment, a heterogeneous network of nodes is formed. Aheterogeneous network of nodes is one in which there are a plurality ofdifferent node types and a plurality of different edge types. Arecommender system then utilizes the heterogeneous network of nodesalong with a single calibrated importance-aware machine learned model torecommend different types of entities in an online network (andpotentially different types of connections as well) to users of theonline network.

DESCRIPTION

The disclosed embodiments provide a method, apparatus, and system forusing a single calibrated importance aware machine learned model torecommend connections of multiple different entity types to a user of anonline network.

As described briefly above, a heterogeneous network of nodes is anetwork in which a plurality of different node types and plurality ofdifferent edge types are permitted. In this context, a “type” refers toa class of items. In the case of nodes, each “type” may represent adifferent class of entities such that each class is represented by adata structure having different fields (in contrast with a datastructure having the same fields but different values in the fields).Thus, for example, one node type may be “user” or “member,” and maydefine a first set of fields that can be used to fill in informationabout a user, such as name, title, place of employment, skills,education, and so forth. A second node type may be a “company” or“organization,” and may define a second set of fields that can be usedto fill in information about an organization, such as industry,location, number of employees, market value, and so forth. A third nodetype may be a newsletter, and may define a set of fields that can beused to fill in information about the newsletter, such as title, author,topic, genre, and so forth. Other examples of node types includehashtags, groups, and events.

In the case of edges, each “type” may represent a different intentionfor action between the corresponding edge. Examples of edge types mayinclude connection edges (where a member generally connects to anothermember, and the intention for action is to permit two-way communicationbetween the members), follow edges (where a member follows ahashtag/company/member, and the intention for action is for the memberto be apprised of changes made to hashtag/company/member nodes orrelated to those nodes), and subscribe edges (where a member subscribesto group/newsletter/event, and the intention for action is for themember to be apprised of updates or new versions of the correspondingnodes).

Each edge may be either bidirectional or unidirectional. Bidirectionaledges provide a link between the nodes that operates in both directions(e.g., from node A to node B and from node B to node A). A connectionedge is an example of a bidirectional edge, as it works to allow theuser corresponding to node A to communicate with the user correspondingto node B and vice-versa). Unidirectional edges provide a link betweennodes that only operates in one direction (e.g., from node A to node Bbut not from node B to node A). Examples of unidirectional edges includesubscribe edges and follow edges.

One way to make recommendations to members of various different entities(of potentially different entity types) is to train a separate machinelearned model on each different combination of node type and edge type.Thus, for example, there may be a first machine learned model trainedfor “member” “connection edges,” a second machine learned model trainedfor “member” “follow edges,” a third machine learned model trained for“hashtag” “follow edges,” and so on. Each of these separately trainedmachine learned models may be called an Edge-FPR model and essentiallyrank different entities of their corresponding node type against eachother.

The Edge-FPR models are separate and independent artificial intelligence(AI) models that can each leverage their own machine learning algorithm,such as XGBoost, Logistic Regression, Deep Neural Networks, and thelike, along with their own domain knowledge, to rank the entities. Theymay also be optimized for different immediate metrics (e.g., clicks) andmight even be generating scores for the entities of the correspondingnode types using different scales. This makes it difficult to comparethe importance of one entity of a first entity type against a secondentity of a second entity type.

Thus, it would be beneficial to have a single machine learned model thatallows for ranking of entities across all the different combinations ofnode types and edge types. One technical challenge in doing so, however,is the aforementioned scale issue—each Edge-FPR model produces scoresthat can be on different scales. As such, in an example embodiment, asolution is provided that calibrates the scores from the Edge-FPR modelsto a single scale. Specifically, log-linear regression is utilized overquantiles of scores from Edge-FPR models to calibrate them and make themcomparable.

An additional technical challenge is that member engagement is differentfor the different edge types. The result is that there can be adifferent training data distribution for each edge, which can result ina bias towards certain edge types (certain edge types having higherimportance than other edge types). As such, in an example embodiment,the solution utilizes a per-edge type multiplicative factor dictated bythe true importance of an edge type, which is learned through acounterfactual experimentation process.

An additional technical challenge in utilizing a single machine learnedmodel is that each Edge-FPR model may be optimized on a differentmetric. Thus, in an example embodiment, the solution optimizes on asingle, common downstream metric, specifically downstream interactions,that can be compared against each other across all combinations of nodetypes and edge types.

FIG. 1 is a block diagram illustrating a client-server system 100, inaccordance with an example embodiment. A networked system 102 providesserver-side functionality via a network 104 (e.g., the Internet or awide area network (WAN)) to one or more clients. FIG. 1 illustrates, forexample, a web client 106 (e.g., a browser) and a programmatic client108 executing on respective client machines 110 and 112.

An application program interface (API) server 114 and a web server 116are coupled to, and provide programmatic and web interfaces respectivelyto, one or more application servers 118. The application server(s) 118host one or more applications 120. The application server(s) 118 are, inturn, shown to be coupled to one or more database servers 124 thatfacilitate access to one or more databases 126. While the application(s)120 are shown in FIG. 1 to form part of the networked system 102, itwill be appreciated that, in alternative embodiments, the application(s)120 may form part of a service that is separate and distinct from thenetworked system 102.

Further, while the client-server system 100 shown in FIG. 1 employs aclient-server architecture, the present disclosure is, of course, notlimited to such an architecture, and could equally well find applicationin a distributed, or peer-to-peer, architecture system, for example. Thevarious applications 120 could also be implemented as standalonesoftware programs, which do not necessarily have networkingcapabilities.

The web client 106 accesses the various applications 120 via the webinterface supported by the web server 116. Similarly, the programmaticclient 108 accesses the various services and functions provided by theapplication(s) 120 via the programmatic interface provided by the APIserver 114.

FIG. 1 also illustrates a third-party application 128, executing on athird-party server 130, as having programmatic access to the networkedsystem 102 via the programmatic interface provided by the API server114. For example, the third-party application 128 may, utilizinginformation retrieved from the networked system 102, support one or morefeatures or functions on a website hosted by a third party. Thethird-party website may, for example, provide one or more functions thatare supported by the relevant applications 120 of the networked system102.

In some embodiments, any website referred to herein may comprise onlinecontent that may be rendered on a variety of devices including, but notlimited to, a desktop personal computer (PC), a laptop, and a mobiledevice (e.g., a tablet computer, smartphone, etc.). In this respect, anyof these devices may be employed by a user to use the features of thepresent disclosure. In some embodiments, a user can use a mobile app ona mobile device (any of the machines 110, 112 and the third-party server130 may be a mobile device) to access and browse online content, such asany of the online content disclosed herein. A mobile server (e.g., APIserver 114) may communicate with the mobile app and the applicationserver(s) 118 in order to make the features of the present disclosureavailable on the mobile device.

In some embodiments, the networked system 102 may comprise functionalcomponents of an online network. FIG. 2 is a block diagram showing thefunctional components of an online network, including a data processingmodule referred to herein as a search engine 216, for use in generatingand providing search results for a search query, consistent with someembodiments of the present disclosure. In some embodiments, the searchengine 216 may reside on the application server(s) 118 in FIG. 1 .However, it is contemplated that other configurations are also withinthe scope of the present disclosure.

As shown in FIG. 2 , a front end may comprise a user interface module(e.g., a web server 116) 212, which receives requests from variousclient computing devices and communicates appropriate responses to therequesting client devices. For example, the user interface module(s) 212may receive requests in the form of Hypertext Transfer Protocol (HTTP)requests or other web-based API requests. In addition, a userinteraction detection module 213 may be provided to detect variousinteractions that users have with different applications 120, services,and content presented. As shown in FIG. 2 , upon detecting a particularinteraction, the user interaction detection module 213 logs theinteraction, including the type of interaction and any metadata relatingto the interaction, in a user activity and behavior database 222.

An application logic layer may include one or more various applicationserver modules 214, which, in conjunction with the user interfacemodule(s) 212, generate various user interfaces (e.g., web pages) withdata retrieved from various data sources in a data layer. In someembodiments, individual application server modules 214 are used toimplement the functionality associated with various applications 120and/or services provided by the online network.

As shown in FIG. 2 , the data layer may include several databases 126,such as a profile database 218 for storing profile data, including bothuser profile data and profile data for various organizations (e.g.,companies, schools, etc.). Consistent with some embodiments, when aperson initially registers to become a user of the online network, theperson will be prompted to provide some personal information, such ashis or her name, age (e.g., birthdate), gender, interests, contactinformation, home town, address, spouse's and/or family members' names,educational background (e.g., schools, majors, matriculation and/orgraduation dates, etc.), employment history, skills, professionalorganizations, and so on. This information is stored, for example, inthe profile database 218. Similarly, when a representative of anorganization initially registers the organization with the onlinenetwork, the representative may be prompted to provide certaininformation about the organization. This information may be stored, forexample, in the profile database 218, or another database (not shown).In some embodiments, the profile data may be processed (e.g., in thebackground or offline) to generate various derived profile data. Forexample, if a user has provided information about various job titlesthat the user has held with the same organization or differentorganizations, and for how long, this information can be used to inferor derive a user profile attribute indicating the user's overallseniority level or seniority level within a particular organization. Insome embodiments, importing or otherwise accessing data from one or moreexternally hosted data sources may enrich profile data for both usersand organizations. For instance, with organizations in particular,financial data may be imported from one or more external data sourcesand made part of an organization's profile. This importation oforganization data and enrichment of the data will be described in moredetail later in this document.

Once registered, a user may invite other users, or be invited by otherusers, to connect via the online network. A “connection” may constitutea bilateral agreement by the users, such that both users acknowledge theestablishment of the connection. Similarly, in some embodiments, a usermay elect to “follow” another user. In contrast to establishing aconnection, the concept of “following” another user typically is aunilateral operation and, at least in some embodiments, does not requireacknowledgement or approval by the user that is being followed. When oneuser follows another, the user who is following may receive statusupdates (e.g., in an activity or content stream) or other messagespublished by the user being followed, relating to various activitiesundertaken by the user being followed. Similarly, when a user follows anorganization, the user becomes eligible to receive messages or statusupdates published on behalf of the organization. For instance, messagesor status updates published on behalf of an organization that a user isfollowing will appear in the user's personalized data feed, commonlyreferred to as an activity stream or content stream. In any case, thevarious associations and relationships that the users establish withother users, or with other entities and objects, are stored andmaintained within a social graph in a social graph database 220.

As users interact with the various applications 120, services, andcontent made available via the online network, the users' interactionsand behavior (e.g., content viewed, links or buttons selected, messagesresponded to, etc.) may be tracked, and information concerning theusers' activities and behavior may be logged or stored, for example, asindicated in FIG. 2 , by the user activity and behavior database 222.This logged activity information may then be used by the search engine216 to determine search results for a search query.

In some embodiments, the databases 218, 220, and 222 may be incorporatedinto the database(s) 126 in FIG. 1 . However, other configurations arealso within the scope of the present disclosure.

Although not shown, in some embodiments, the social networking system210 provides an API module via which applications 120 and services canaccess various data and services provided or maintained by the onlinenetwork. For example, using an API, an application may be able torequest and/or receive one or more recommendations. Such applications120 may be browser-based applications 120 or may be operatingsystem-specific. In particular, some applications 120 may reside andexecute (at least partially) on one or more mobile devices (e.g., phoneor tablet computing devices) with a mobile operating system.Furthermore, while in many cases the applications 120 or services thatleverage the API may be applications 120 and services that are developedand maintained by the entity operating the online network, nothing otherthan data privacy concerns prevents the API from being provided to thepublic or to certain third parties under special arrangements, therebymaking the navigation recommendations available to third-partyapplications 128 and services.

Although features of the present disclosure are referred to herein asbeing used or presented in the context of a web page, it is contemplatedthat any user interface view (e.g., a user interface on a mobile deviceor on desktop software) is within the scope of the present disclosure.

In an example embodiment, when user profiles are indexed, forward searchindexes are created and stored. The search engine 216 facilitates theindexing and searching for content within the online network, such asthe indexing and searching for data or information contained in the datalayer, such as profile data (stored, e.g., in the profile database 218),social graph data (stored, e.g., in the social graph database 220), anduser activity and behavior data (stored, e.g., in the user activity andbehavior database 222). The search engine 216 may collect, parse, and/orstore data in an index or other similar structure to facilitate theidentification and retrieval of information in response to receivedqueries for information. This may include, but is not limited to,forward search indexes, inverted indexes, N-gram indexes, and so on.

FIG. 3 is a block diagram illustrating application server module 214 ofFIG. 2 in more detail, in accordance with an example embodiment. Whilein many embodiments the application server module 214 will contain manysubcomponents used to perform various different actions within thesocial networking system 210, in FIG. 3 only those components that arerelevant to the present disclosure are depicted.

A heterogonous graph generator 300 may create a heterogeneous network ofnodes from social network information from the databases 218, 220, 222.The process involved in creating the heterogonous network of nodes isbeyond the scope of this disclosure, but generally involves taking datastructures representing entities of different node types and formingthese data structures into nodes of different types, then connecting thenodes via edges of potentially different types based on various types ofconnections defined between the data structures.

A plurality of different Edge-FPR models 302A-302N (one for eachdifferent combination of node type and edge type) may each utilize theheterogeneous network of nodes and an input entity to score each of aplurality of different possible entities of their corresponding entitytype. The score is a relative score based on a comparison of each of theplurality of different possible entities of their corresponding entitytype against each other, and these scores may be on a different scaleand optimized for different things based upon the Edge-FPR model302A-302N. These scores may represent a likelihood that the input entitywould be interested in forming a connection with the correspondingentity from the plurality of different possible entities (that interestbeing possibly measured based on different metrics based on the Edge-FPRmodel in question).

Each of the Edge FPR models 302A-302N may be trained by a machinelearning algorithm (potentially different machine learning algorithm) tomake their predictions. This training may include specifying anoptimization parameter, such as a metric like downstream interaction byusers within a first time period (e.g., thirty days) and then feedingtraining data, which may include sample entities and sample users andtheir corresponding interactions to the corresponding machine learningalgorithm. The machine learning algorithm then progressively learnsweights to be applied to each of a plurality of different featuresassociated with the sample users and the sample entities. These weightscan then be applied to values of features of users and entities atevaluation time in order to produce the predictions.

The input entity is simply an entity in the heterogeneous network ofnodes that is being evaluated. This may occur when, for example, theinput entity logs into the online network and navigates to a page orportion of the graphical user interface where recommendations ofentities are to be displayed. Examples of such graphical user interfaceswill be provided in more detail below.

A calibration component 304 then takes each of the individual scoresproduced by the Edge-FPR models 302A-302N and calibrates them based ontheir corresponding Edge-FPR models. As will be described in more detailbelow, calibration itself is performed by a machine learned model,different than the machine learned models that comprise the Edge-FPRmodels 302A-302N.

The output of the calibration component 304 is then calibrated scores.These calibrated scores may then be utilized by a single calibratedimportance-aware machine learned model 306 that is trained to utilizethe calibrated scores, and information about the corresponding entitytypes, to produce a downstream interaction score for each of theplurality of entities of the different entity types. This downstreaminteraction score is then used by a recommendation component 308 to makeone or more recommendations for the input entity. This may includedisplaying the one or more recommendations via a graphical userinterface displayed on a display of a device operated by the inputentity.

In an example embodiment, the single calibrated importance-aware machinelearned model 306 predicts the probability of downstream interactionfrom top-k entities within a cohort and then ranks the cohorts usingthis probability score. A cohort may be defined as a combination of anode type (entity type) and edge type (connection type). Thus, a firstcohort is a first combination of a node type and an entity type while asecond cohort would be a different combination of a node type and anentity type. For example, the following are three different cohorts: (1)User A/connection edge type; (2) User A/follow edge type; (3) UserB/follow edge type. In an example embodiment, the single calibratedimportance-aware machine learned model 306 is a logistic regressionmodel. The model equation may be as follows:

For a member i, let y_(ic) be 1 if the member interacts (likes,comments, shares) with any of the top-k entities from a cohort c,otherwise 0. Then our model equation is:

${P\left( {{y_{ic} = \left. 1 \middle| {c{was}{shown}{to}i} \right.},\theta} \right)} = {{\sum\limits_{Edge}{\theta_{Edge}{\Gamma\left( {f_{Edge}\left( e_{ij}^{Edge} \right)} \right)}}} + {\sum\limits_{k}{\theta_{k}x_{ic}^{k}}}}$

In the above equation, ƒ_(Edge) represents a calibration functionlearned for the corresponding cohort by the calibration component 304. Γis an aggregation function that takes the Edge-FPR scores e_(ij) ^(Edge)for the entities j in cohort c and aggregates them to generate a scoreat the cohort level. Basically, Γ is an aggregation function thatconverts entity-level Edge-FPR scores to cohort level scores.Cohort-level means that the cohort itself is scored, as opposed to theindividual entity within the cohort. x_(ic) ^(k) are additionalfeatures, such as type of cohort, intent of member i, and so forth.θ={θ_(Edge), θ_(k)} are the unknown parameters to be estimated.

Overall, the above model equation utilizes aggregated and calibratedEdge-FPR scores as features in addition to other member/cohort featuresfor estimating the probability of downstream interactions of member i onany top-k entity from cohort c. In an example embodiment, theaggregation function Γ is a linear aggregation function such as weightedaverage. Here, if e_(ij) is the score for an item j in cohort c, thenthe mean aggregation function Γ is given by

$\frac{1}{J}{\sum\limits_{i}{e_{ij}.}}$In other example embodiments, more complicated, nonlinear aggregationfunctions may be utilized.

After scoring the cohort c using the above equation, in an exampleembodiment, this cohort-level score may be multiplied by an importancefactor for the cohort. The cohorts are then ranked using thesemultiplied scores.

Member behavior varies with different edge types. Members like toconnect to people but there are only so many people a member knows. Onthe other hand, a member can follow a lot of hashtags to consume contentrelated to the hashtag. Consequently, this may lead tooverrepresentation of follows edges training data used to train theabove model equation, and thus may result in disproportionally higherscores for follows edges over, for example, connection edges.

In order to address this technical issue, an importance value of aparticular edge type may be estimated by running counterfactualexperiments. Specifically, certain edge types are dropped from amember's heterogeneous network and the impact to engagement of themember is measured. The result is a measurement of how important thoseedge types are, over many members. This measurement is exactly theimportance factor mentioned earlier for a cohort, which is used tomultiply by the cohort-level score.

Calibration performed by the calibration component 304 may be performedby a machine learned function mapping from quantiles of edge-FPR scoresto a quantized observed response. Edge-FPR models use the observednumber of downstream interaction (in some form) as the response fortraining their models. A click, like, comment, or share are examples ofsuch downstream interaction. So for a connection-edge, this would meanthat the number of clicks, likes, comments, or shares on the contentposted by the connected entity would be the downstream interaction whilefor a follows-edge, this would mean that the number of clicks, likes,comments, or shares on content related to the hashtag or companyfollowed would be the downstream interaction.

Let s be the score quartiles generated by an Edge-FPR model for an inputentity and an item (entity from the plurality of possible entities torecommend). Let z be a response variable for this Edge-FPR, and ƒ_(edge)be the calibration function, which is a calibration mapping functionparameterized by a set of unknown parameters Φ, then mathematically:z|Φ˜ƒ_(edge)(s)

The following equation estimates a mapping function ƒ_(edge) by runninga log-linear regression, namely ƒ_(edge), that takes the following form:z|ϕ˜g⁻¹(w₁s⁰+w₂s¹+w₃s²+w₄s³+w₅ log(1+s)+w₆(1+s)⁻¹)where:

-   -   z is Edge-FPR model's response (for example, number of likes by        a viewer on an item),    -   g is the log link function,    -   ϕ={w₁, w₂, w₃, w₄, w₅, w₆} is the set of unknown weights to be        estimated,    -   s is the score quartile for viewer and cohort, and

$s^{0},s^{1},s^{2},s^{3},{\log\left( {1 + s} \right)},\frac{1}{\left( {1 + s} \right)}$

-   -    are the transformations of s used as regressors.

In an example embodiment, a common downstream metric is used as thecommon baseline to evaluate the different edge types. This is importantas it helps ensure that the system is moving towards the eventual goalof optimizing members' engagement and helps define the objective of thesingle model. In an example embodiment, this common downstream metric isactive edges. An active edge is defined as a viral interaction, such asa like, comment, or share within 7 days of edge creation for an entity.For connection edge types, this definition is expanded to includecommunication, such as private messages. Notably, this interaction timeperiod (7 days) is different than the 30 day time period used by one ormore of the Edge FPR models.

FIG. 4 is a flow diagram illustrating a method 400 for utilizing aplurality of machine-learned models, in accordance with an exampleembodiment. At operation 402, an identification of a first user of anonline network is received. Then, a loop is performed for each of aplurality of different cohorts in an online network. As described above,a cohort is a unique combination of a type of entity (e.g., user,organization, hashtag, event, etc.) and type of connection (e.g.,regular connection, such as between users, follow connection, subscribeconnection) within the online network. At operation 404, scores for aplurality of different entities of the corresponding cohort are obtainedfrom a cohort-specific machine learned prediction model corresponding tothe cohort. These scores are on a scale corresponding to the cohort andare indicative of the likelihood that the first user will interact witheach of the plurality of different entities. It should be noted that,for purposes of this disclosure, interacting with an entity involves anaction taken within a graphical user interface to interact with someaspect of the entity and can include interacting with content posted bythe entity or content posted about the entity, and should not be limitedto directly interacting with the entity itself. In an exampleembodiment, this likelihood is specifically the likelihood that thefirst user will interact with each of the plurality of differententities within a first time period.

At operation 406, the scores for the plurality of different entities arecalibrated onto a common scale. The common scale is a scale that will beused for multiple cohort-specific machine learned prediction models,rather than just a single cohort-specific machine learned predictionmodel. At operation 408, it is determined whether there are any morecohorts to examine. If so, then the method 400 loops back to operation404 for the next cohort. If not, then at operation 410, theidentification of a first user and the calibrated scores from theplurality of different cohort-specific machine learned prediction modelsare fed into a single calibrated importance-aware machine learned modeltrained to predict a probability of downstream interaction by users, viaa graphical user interface, with top-k entities within each of a firstplurality of cohorts and ranking the plurality of cohorts, thusresulting in the first cohort and the second cohort being ranked for thefirst user. At operation 412, one or more entities of each of theplurality of cohorts is displayed, in the graphical user interface, tothe first user, in a manner that all entities of any given cohort aredisplayed above all entities of any lower ranking cohort.

FIG. 5 is a block diagram illustrating a screen capture of a graphicaluser interface 500, in accordance with an example embodiment. Thisdiagram illustrates an example output of operation 412 of FIG. 4 .Specifically, there are four cohorts of entities depicted as recommendedconnections for a first user 502. These cohorts include people (forconnection) 504, organizations 506, people (for follow) 508, newsletters510, and groups 512. Notably, all of the recommendations for people (forconnection) are displayed above all of the recommendations fororganizations, as well as above the other cohorts. This indicates thatthe single calibrated importance-aware machine learned model has rankedthe people (for connection) cohort higher than the organization cohortand all the other cohorts for the first user 502.

FIG. 6 is a block diagram 600 illustrating a software architecture 602,which can be installed on any one or more of the devices describedabove. FIG. 6 is merely a non-limiting example of a softwarearchitecture, and it will be appreciated that many other architecturescan be implemented to facilitate the functionality described herein. Invarious embodiments, the software architecture 602 is implemented byhardware such as a machine 700 of FIG. 7 that includes processors 710,memory 730, and input/output (I/O) components 750. In this examplearchitecture, the software architecture 602 can be conceptualized as astack of layers where each layer may provide a particular functionality.For example, the software architecture 602 includes layers such as anoperating system 604, libraries 606, frameworks 608, and applications610. Operationally, the applications 610 invoke API calls 612 throughthe software stack and receive messages 614 in response to the API calls612, consistent with some embodiments.

In various implementations, the operating system 604 manages hardwareresources and provides common services. The operating system 604includes, for example, a kernel 620, services 622, and drivers 624. Thekernel 620 acts as an abstraction layer between the hardware and theother software layers, consistent with some embodiments. For example,the kernel 620 provides memory management, processor management (e.g.,scheduling), component management, networking, and security settings,among other functionality. The services 622 can provide other commonservices for the other software layers. The drivers 624 are responsiblefor controlling or interfacing with the underlying hardware, accordingto some embodiments. For instance, the drivers 624 can include displaydrivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers,flash memory drivers, serial communication drivers (e.g., UniversalSerial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, powermanagement drivers, and so forth.

In some embodiments, the libraries 606 provide a low-level commoninfrastructure utilized by the applications 610. The libraries 606 caninclude system libraries 630 (e.g., C standard library) that can providefunctions such as memory allocation functions, string manipulationfunctions, mathematic functions, and the like. In addition, thelibraries 606 can include API libraries 632 such as media libraries(e.g., libraries to support presentation and manipulation of variousmedia formats such as Moving Picture Experts Group-4 (MPEG4), AdvancedVideo Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3),Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec,Joint Photographic Experts Group (JPEG or JPG), or Portable NetworkGraphics (PNG)), graphics libraries (e.g., an OpenGL framework used torender in two dimensions (2D) and three dimensions (3D) in a graphiccontext on a display), database libraries (e.g., SQLite to providevarious relational database functions), web libraries (e.g., WebKit toprovide web browsing functionality), and the like. The libraries 606 canalso include a wide variety of other libraries 634 to provide many otherAPIs to the applications 610.

The frameworks 608 provide a high-level common infrastructure that canbe utilized by the applications 610, according to some embodiments. Forexample, the frameworks 608 provide various graphical user interfacefunctions, high-level resource management, high-level location services,and so forth. The frameworks 608 can provide a broad spectrum of otherAPIs that can be utilized by the applications 610, some of which may bespecific to a particular operating system 604 or platform.

In an example embodiment, the applications 610 include a homeapplication 650, a contacts application 652, a browser application 654,a book reader application 656, a location application 658, a mediaapplication 660, a messaging application 662, a game application 664,and a broad assortment of other applications, such as a third-partyapplication 666. According to some embodiments, the applications 610 areprograms that execute functions defined in the programs. Variousprogramming languages can be employed to create one or more of theapplications 610, structured in a variety of manners, such asobject-oriented programming languages (e.g., Objective-C, Java, or C++)or procedural programming languages (e.g., C or assembly language). In aspecific example, the third-party application 666 (e.g., an applicationdeveloped using the ANDROID™ or IOS™ software development kit (SDK) byan entity other than the vendor of the particular platform) may bemobile software running on a mobile operating system such as IOS™,ANDROID™, WINDOWS® Phone, or another mobile operating system. In thisexample, the third-party application 666 can invoke the API calls 612provided by the operating system 604 to facilitate functionalitydescribed herein.

FIG. 7 illustrates a diagrammatic representation of a machine 700 in theform of a computer system within which a set of instructions may beexecuted for causing the machine 700 to perform any one or more of themethodologies discussed herein, according to an example embodiment.Specifically, FIG. 7 shows a diagrammatic representation of the machine700 in the example form of a computer system, within which instructions716 (e.g., software, a program, an application 610, an applet, an app,or other executable code) for causing the machine 700 to perform any oneor more of the methodologies discussed herein may be executed. Forexample, the instructions 716 may cause the machine 700 to execute themethod 400 of FIG. 4 . Additionally, or alternatively, the instructions716 may implement FIGS. 1-5 , and so forth. The instructions 716transform the general, non-programmed machine 700 into a particularmachine 700 programmed to carry out the described and illustratedfunctions in the manner described. In alternative embodiments, themachine 700 operates as a standalone device or may be coupled (e.g.,networked) to other machines. In a networked deployment, the machine 700may operate in the capacity of a server machine or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine 700 maycomprise, but not be limited to, a server computer, a client computer, aPC, a tablet computer, a laptop computer, a netbook, a set-top box(STB), a portable digital assistant (PDA), an entertainment mediasystem, a cellular telephone, a smartphone, a mobile device, a wearabledevice (e.g., a smart watch), a smart home device (e.g., a smartappliance), other smart devices, a web appliance, a network router, anetwork switch, a network bridge, or any machine capable of executingthe instructions 716, sequentially or otherwise, that specify actions tobe taken by the machine 700. Further, while only a single machine 700 isillustrated, the term “machine” shall also be taken to include acollection of machines 700 that individually or jointly execute theinstructions 716 to perform any one or more of the methodologiesdiscussed herein.

The machine 700 may include processors 710, memory 730, and I/Ocomponents 750, which may be configured to communicate with each othersuch as via a bus 702. In an example embodiment, the processors 710(e.g., a central processing unit (CPU), a reduced instruction setcomputing (RISC) processor, a complex instruction set computing (CISC)processor, a graphics processing unit (GPU), a digital signal processor(DSP), an application-specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), another processor, or anysuitable combination thereof) may include, for example, a processor 712and a processor 714 that may execute the instructions 716. The term“processor” is intended to include multi-core processors 710 that maycomprise two or more independent processors 712 (sometimes referred toas “cores”) that may execute instructions 716 contemporaneously.Although FIG. 7 shows multiple processors 710, the machine 700 mayinclude a single processor 712 with a single core, a single processor712 with multiple cores (e.g., a multi-core processor), multipleprocessors 710 with a single core, multiple processors 710 with multiplecores, or any combination thereof.

The memory 730 may include a main memory 732, a static memory 734, and astorage unit 736, all accessible to the processors 710 such as via thebus 702. The main memory 732, the static memory 734, and the storageunit 736 store the instructions 716 embodying any one or more of themethodologies or functions described herein. The instructions 716 mayalso reside, completely or partially, within the main memory 732, withinthe static memory 734, within the storage unit 736, within at least oneof the processors 710 (e.g., within the processor's cache memory), orany suitable combination thereof, during execution thereof by themachine 700.

The I/O components 750 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 750 that are included in a particular machine 700 will dependon the type of machine 700. For example, portable machines such asmobile phones will likely include a touch input device or other suchinput mechanisms, while a headless server machine will likely notinclude such a touch input device. It will be appreciated that the I/Ocomponents 750 may include many other components that are not shown inFIG. 7 . The I/O components 750 are grouped according to functionalitymerely for simplifying the following discussion, and the grouping is inno way limiting. In various example embodiments, the I/O components 750may include output components 752 and input components 754. The outputcomponents 752 may include visual components (e.g., a display such as aplasma display panel (PDP), a light-emitting diode (LED) display, aliquid crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), haptic components (e.g., avibratory motor, resistance mechanisms), other signal generators, and soforth. The input components 754 may include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point-based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or another pointinginstrument), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

In further example embodiments, the I/O components 750 may includebiometric components 756, motion components 758, environmentalcomponents 760, or position components 762, among a wide array of othercomponents. For example, the biometric components 756 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram-basedidentification), and the like. The motion components 758 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environmental components 760 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detect concentrations of hazardous gases for safetyor to measure pollutants in the atmosphere), or other components thatmay provide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 762 mayinclude location sensor components (e.g., a Global Positioning System(GPS) receiver component), altitude sensor components (e.g., altimetersor barometers that detect air pressure from which altitude may bederived), orientation sensor components (e.g., magnetometers), and thelike.

Communication may be implemented using a wide variety of technologies.The I/O components 750 may include communication components 764 operableto couple the machine 700 to a network 780 or devices 770 via a coupling782 and a coupling 772, respectively. For example, the communicationcomponents 764 may include a network interface component or anothersuitable device to interface with the network 780. In further examples,the communication components 764 may include wired communicationcomponents, wireless communication components, cellular communicationcomponents, near field communication (NFC) components, Bluetooth®components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and othercommunication components to provide communication via other modalities.The devices 770 may be another machine or any of a wide variety ofperipheral devices (e.g., a peripheral device coupled via a USB).

Moreover, the communication components 764 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 764 may include radio frequency identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components764, such as location via Internet Protocol (IP) geolocation, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

Executable Instructions and Machine Storage Medium

The various memories (i.e., 730, 732, 734, and/or memory of theprocessor(s) 710) and/or the storage unit 736 may store one or more setsof instructions 716 and data structures (e.g., software) embodying orutilized by any one or more of the methodologies or functions describedherein. These instructions (e.g., the instructions 716), when executedby the processor(s) 710, cause various operations to implement thedisclosed embodiments.

As used herein, the terms “machine-storage medium,” “device-storagemedium,” and “computer-storage medium” mean the same thing and may beused interchangeably. The terms refer to a single or multiple storagedevices and/or media (e.g., a centralized or distributed database,and/or associated caches and servers) that store executable instructions716 and/or data. The terms shall accordingly be taken to include, butnot be limited to, solid-state memories, and optical and magnetic media,including memory internal or external to the processors 710. Specificexamples of machine-storage media, computer-storage media, and/ordevice-storage media include non-volatile memory including, by way ofexample, semiconductor memory devices, e.g., erasable programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM), field-programmable gate array (FPGA), and flash memorydevices; magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms“machine-storage media,” “computer-storage media,” and “device-storagemedia” specifically exclude carrier waves, modulated data signals, andother such media, at least some of which are covered under the term“signal medium” discussed below.

Transmission Medium

In various example embodiments, one or more portions of the network 780may be an ad hoc network, an intranet, an extranet, a VPN, a LAN, aWLAN, a WAN, a WWAN, a MAN, the Internet, a portion of the Internet, aportion of the PSTN, a plain old telephone service (POTS) network, acellular telephone network, a wireless network, a Wi-Fi® network,another type of network, or a combination of two or more such networks.For example, the network 780 or a portion of the network 780 may includea wireless or cellular network, and the coupling 782 may be a CodeDivision Multiple Access (CDMA) connection, a Global System for Mobilecommunications (GSM) connection, or another type of cellular or wirelesscoupling. In this example, the coupling 782 may implement any of avariety of types of data transfer technology, such as Single CarrierRadio Transmission Technology (1×RTT), Evolution-Data Optimized (EVDO)technology, General Packet Radio Service (GPRS) technology, EnhancedData rates for GSM Evolution (EDGE) technology, third GenerationPartnership Project (3GPP) including 3G, fourth generation wireless (4G)networks, Universal Mobile Telecommunications System (UMTS), High-SpeedPacket Access (HSPA), Worldwide Interoperability for Microwave Access(WiMAX), Long-Term Evolution (LTE) standard, others defined by variousstandard-setting organizations, other long-range protocols, or otherdata-transfer technology.

The instructions 716 may be transmitted or received over the network 780using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components764) and utilizing any one of a number of well-known transfer protocols(e.g., HTTP). Similarly, the instructions 716 may be transmitted orreceived using a transmission medium via the coupling 772 (e.g., apeer-to-peer coupling) to the devices 770. The terms “transmissionmedium” and “signal medium” mean the same thing and may be usedinterchangeably in this disclosure. The terms “transmission medium” and“signal medium” shall be taken to include any intangible medium that iscapable of storing, encoding, or carrying the instructions 716 forexecution by the machine 700, and include digital or analogcommunications signals or other intangible media to facilitatecommunication of such software. Hence, the terms “transmission medium”and “signal medium” shall be taken to include any form of modulated datasignal, carrier wave, and so forth. The term “modulated data signal”means a signal that has one or more of its characteristics set orchanged in such a manner as to encode information in the signal.

Computer-Readable Medium

The terms “machine-readable medium,” “computer-readable medium,” and“device-readable medium” mean the same thing and may be usedinterchangeably in this disclosure. The terms are defined to includeboth machine-storage media and transmission media. Thus, the termsinclude both storage devices/media and carrier waves/modulated datasignals.

What is claimed is:
 1. A system comprising: a non-transitorycomputer-readable medium having instructions stored thereon, which, whenexecuted by a processor, cause the system to perform operationscomprising: training a first cohort-specific machine learned pre dictionmodel corresponding to a first cohort by specifying a first optimizationparameter and then feeding first training data and the firstoptimization parameter into a first machine learning algorithm, thefirst training data including sample entities of a first type of entityin an online network and sample entries of a first type of connectionbetween entities in the online network; receiving an identification of afirst user in an online network; obtaining, from the firstcohort-specific machine learned prediction model, first scores for aplurality of different entities of the first cohort based on theidentification of the first user, the first scores indicative of alikelihood of the first user interacting with each of the plurality ofdifferent entities of the first cohort via a graphical user interface ofthe online network, wherein the first cohort is a first combination ofthe first type of entity in an online network and the first type ofconnection between entities in the online network, the firstcohort-specific machine learned prediction model calculating firstscores on a first scale; calibrating the first scores onto a commonscale utilized by multiple cohort-specific machine learned predictionmodels; obtaining, from a second cohort-specific machine learnedprediction model corresponding to a second cohort based on theidentification of the first user, second scores for a plurality ofdifferent entities of the second cohort, the second scores indicative ofa likelihood of the first user interacting with each of the plurality ofdifferent entities of the second cohort via a graphical user interfaceof the online network, wherein the second cohort is a second combinationof the first type of entity in the online network and a second type ofconnection between entities in the online network, the secondcohort-specific machine learned prediction model calculating the secondscores on a second scale, wherein the second cohort-specific machinelearned prediction model is trained separately from the firstcohort-specific machine learned prediction model; calibrating the secondscores onto the common scale; feeding the identification of the firstuser, the calibrated first scores, and the calibrated second scores intoa single calibrated importance-aware machine learned model trained topredict a probability of downstream interaction by users with top-kentities within each of a plurality of cohorts to obtain a ranking ofthe plurality of cohorts for the first user; and causing display, in thegraphical user interface, to the first user, of one or more entities ofthe first cohort and one or more entities of the second cohort in amanner that entities of a higher ranking cohort are displayed aboveentities of a lower ranking cohort.
 2. The system of claim 1, whereinthe first cohort-specific machine learned prediction model is a neuralnetwork.
 3. The system of claim 1, wherein the single calibratedimportance-aware machine learned model converts entity-level scores tocohort-level scores using an aggregation function.
 4. The system ofclaim 3, wherein the aggregation function is a weighted average.
 5. Thesystem of claim 3, wherein the aggregation function is a non-linearaggregation function.
 6. The system of claim 3, wherein eachcohort-level score is multiplied by a different importance factor uniqueto the corresponding cohort, wherein the importance factor for eachcohort is estimated by dropping edges corresponding to the correspondingcohort from member heterogonous graphs of entities in training data andmeasuring impact to the downstream interaction of members.
 7. The systemof claim 1, wherein the calibrating is performed by machine learnedfunction mapping from quantiles of scores from cohort-specific machinelearned models to a quantized observed response.
 8. The system of claim7, wherein the quantized observed response is a click, like, comment, orshare action in a graphical user interface.
 9. The system of claim 1,wherein the operations further comprise: training the secondcohort-specific machine learned prediction model by specificizing asecond optimization parameter and then feeding second training data andthe second optimization parameter into a second machine learningalgorithm different than the first machine learning algorithm, thesecond training data including sample entities of the first type ofentity and sample entries of the second type of connection.
 10. Thesystem of claim 1, wherein the operations further comprise: dynamicallyadjusting how the one or more entities of the first cohort and the oneor more entities of the second cohort are displayed in the graphicaluser interface in response to a second execution of the obtaining from asecond cohort-specific machine learned prediction model corresponding toa second cohort, scores for a plurality of different entities of thesecond cohort.
 11. A computerized method comprising: training a firstcohort-specific machine learned pre diction model corresponding to afirst cohort by specifying a first optimization parameter and thenfeeding first training data and the first optimization parameter into afirst machine learning algorithm, the first training data includingsample entities of a first type of entity in an online network andsample entries of a first type of connection between entities in theonline network; receiving an identification of a first user in an onlinenetwork; obtaining, from the first cohort-specific machine learnedprediction model, first scores for a plurality of different entities ofthe first cohort based on the identification of the first user, thefirst scores indicative of a likelihood of the first user interactingwith each of the plurality of different entities of the first cohort viaa graphical user interface of the online network, wherein the firstcohort is a first combination of the first type of entity in an onlinenetwork and the first type of connection between entities in the onlinenetwork, the first cohort-specific machine learned prediction modelcalculating first scores on a first scale; calibrating the first scoresonto a common scale utilized by multiple cohort-specific machine learnedprediction models; obtaining, from a second cohort-specific machinelearned prediction model corresponding to a second cohort based on theidentification of the first user, second scores for a plurality ofdifferent entities of the second cohort, the second scores indicative ofa likelihood of the first user interacting with each of the plurality ofdifferent entities of the second cohort via a graphical user interfaceof the online network, wherein the second cohort is a second combinationof the first type of entity in the online network and a second type ofconnection between entities in the online network, the secondcohort-specific machine learned prediction model calculating the secondscores on a second scale, wherein the second cohort-specific machinelearned prediction model is trained separately from the firstcohort-specific machine learned prediction model; calibrating the secondscores onto the common scale; feeding the identification of the firstuser, the calibrated first scores, and the calibrated second scores intoa single calibrated importance-aware machine learned model trained topredict a probability of downstream interaction by users with top-kentities within each of a plurality of cohorts to obtain a ranking ofthe plurality of cohorts for the first user; and causing display, in thegraphical user interface, to the first user, of one or more entities ofthe first cohort and one or more entities of the second cohort in amanner that entities of a higher ranking cohort are displayed aboveentities of a lower ranking cohort.
 12. The method of claim 11, whereinthe first cohort-specific machine learned prediction model is a neuralnetwork.
 13. The method of claim 11, wherein the single calibratedimportance-aware machine learned model converts entity-level scores tocohort-level scores using an aggregation function.
 14. The method ofclaim 13, wherein the aggregation function is a weighted average. 15.The method of claim 13, wherein the aggregation function is a non-linearaggregation function.
 16. The method of claim 13, wherein eachcohort-level score is multiplied by a different importance factor uniqueto the corresponding cohort, wherein the importance factor for eachcohort is estimated by dropping edges corresponding to the correspondingcohort from member heterogonous graphs of entities in training data andmeasuring impact to the downstream interaction of members.
 17. Themethod of claim 11, wherein the calibrating is performed by machinelearned function mapping from quantiles of scores from cohort-specificmachine learned models to a quantized observed response.
 18. The methodof claim 17, wherein the quantized observed response is a click, like,comment, or share action in a graphical user interface.
 19. Anon-transitory machine-readable storage medium comprising instructionswhich, when implemented by one or more machines, cause the one or moremachines to perform operations comprising: training a firstcohort-specific machine learned pre diction model corresponding to afirst cohort by specifying a first optimization parameter and thenfeeding first training data and the first optimization parameter into afirst machine learning algorithm, the first training data includingsample entities of a first type of entity in an online network andsample entries of a first type of connection between entities in theonline network; receiving an identification of a first user in an onlinenetwork; obtaining, from the first cohort-specific machine learnedprediction model, first scores for a plurality of different entities ofthe first cohort based on the identification of the first user, thefirst scores indicative of a likelihood of the first user interactingwith each of the plurality of different entities of the first cohort viaa graphical user interface of the online network, wherein the firstcohort is a first combination of the first type of entity in an onlinenetwork and the first type of connection between entities in the onlinenetwork, the first cohort-specific machine learned prediction modelcalculating first scores on a first scale; calibrating the first scoresonto a common scale utilized by multiple cohort-specific machine learnedprediction models; obtaining, from a second cohort-specific machinelearned prediction model corresponding to a second cohort based on theidentification of the first user, second scores for a plurality ofdifferent entities of the second cohort, the second scores indicative ofa likelihood of the first user interacting with each of the plurality ofdifferent entities of the second cohort via a graphical user interfaceof the online network, wherein the second cohort is a second combinationof the first type of entity in the online network and a second type ofconnection between entities in the online network, the secondcohort-specific machine learned prediction model calculating the secondscores on a second scale, wherein the second cohort-specific machinelearned prediction model is trained separately from the firstcohort-specific machine learned prediction model; calibrating the secondscores onto the common scale; feeding the identification of the firstuser, the calibrated first scores, and the calibrated second scores intoa single calibrated importance-aware machine learned model trained topredict a probability of downstream interaction by users with top-kentities within each of a plurality of cohorts to obtain a ranking ofthe plurality of cohorts for the first user; and causing display, in thegraphical user interface, to the first user, of one or more entities ofthe first cohort and one or more entities of the second cohort in amanner that entities of a higher ranking cohort are displayed aboveentities of a lower ranking cohort.
 20. The non-transitorymachine-readable storage medium of claim 19, wherein the firstcohort-specific machine learned prediction model is a neural network.21. The non-transitory machine-readable storage medium of claim 19,wherein the single calibrated importance-aware machine learned modelconverts entity-level scores to cohort-level scores using an aggregationfunction.
 22. The non-transitory machine-readable storage medium ofclaim 21, wherein the aggregation function is a weighted average.